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Highlights 

Highlights  of  GAO-09-841 ,  a  report  to 
congressional  requesters 


Why  GAO  Did  This  Study 

The  Department  of  Defense  (DOD) 
has  long  been  challenged  in 
effectively  implementing  key 
acquisition  management  controls 
on  its  thousands  of  business 
system  investments.  For  this  and 
other  reasons,  GAO  has  designated 
DOD’s  business  systems 
modernization  efforts  as  high-risk 
since  1995.  One  major  business 
system  investment  is  the  Navy’s 
Enterprise  Resource  Planning 
(ERP)  system.  Initiated  in  2003,  it  is 
to  standardize  the  Navy’s  business 
processes,  such  as  acquisition  and 
financial  management.  It  is  being 
delivered  in  increments,  the  first  of 
which  is  to  cost  about  $2.4  billion 
over  its  20-year  useful  life  and  be 
fully  deployed  by  fiscal  year  2013. 
To  date,  the  program  has 
experienced  about  $570  million  in 
cost  overruns  and  a  2-year 
schedule  delay.  GAO  was  asked  to 
determine  whether  (1)  system 
testing  is  being  effectively 
managed,  (2)  system  changes  are 
being  effectively  controlled,  and 
(3)  independent  verification  and 
validation  (IV&V)  activities  are 
being  effectively  managed.  To  do 
this,  GAO  analyzed  relevant 
program  documentation,  traced 
random  samples  of  test  defects  and 
change  requests,  and  interviewed 
cognizant  officials. 


What  GAO  Recommends 


GAO  is  making  recommendations 
to  the  Secretary  of  Defense  aimed 
at  improving  the  program’s  system 
change  request  review  and 
approval  process  and  its  IV&V 
activities.  DOD  concurred  with  the 
recommendations  and  identified 
actions  that  it  plans  to  take. 

View  GAO-09-841  or  key  components. 

For  more  information,  contact  Randolph  C. 
Hite  at  (202)  512-3439  or  hiter@gao.gov. 
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Navy  Implementing  a  Number  of  Key  Management 
Controls  on  Enterprise  Resource  Planning  System, 
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What  GAO  Found 

The  Navy  has  largely  implemented  effective  controls  on  Navy  ERP  associated 
with  system  testing  and  change  control.  For  example,  it  has  established  a  well- 
defined  structure  for  managing  tests,  including  providing  for  a  logical 
sequence  of  test  events,  adequately  planning  key  test  events,  and  documenting 
and  reporting  test  results.  In  addition,  it  has  documented,  and  is  largely 
following,  its  change  request  review  and  approval  process,  which  reflects  key 
aspects  of  relevant  guidance,  such  as  having  defined  roles  and  responsibilities 
and  a  hierarchy  of  control  boards.  However,  important  aspects  of  test 
management  and  change  control  have  not  been  fully  implemented. 

Specifically,  the  program’s  tool  for  auditing  defect  management  did  not 
always  record  key  data  about  changes  made  to  the  status  of  identified  defects. 
To  its  credit,  the  program  office  recently  took  steps  to  address  this,  thereby 
reducing  the  risk  of  defect  status  errors  or  unauthorized  changes.  Also,  while 
the  program  office’s  change  review  and  approval  procedures  include 
important  steps,  such  as  considering  the  impact  of  a  change,  and  program 
officials  told  GAO  that  cost  and  schedule  impacts  of  a  change  are  discussed  at 
control  board  meetings,  GAO’s  analysis  of  60  randomly  selected  change 
requests  showed  no  evidence  that  cost  and  schedule  impacts  were  in  fact 
considered.  Without  such  key  information,  decision-making  authorities  lack 
an  adequate  basis  for  making  informed  investment  decisions,  which  could 
result  in  cost  overruns  and  schedule  delays. 

The  Navy  has  not  effectively  managed  its  IV&V  activities,  which  are  designed 
to  obtain  an  unbiased  position  on  whether  product  and  process  standards  are 
being  met.  In  particular,  the  Navy  has  not  ensured  that  the  IV&V  contractor  is 
independent  of  the  products  and  processes  that  it  is  reviewing.  Specifically, 
the  same  contractor  responsible  for  performing  IV&V  of  Navy  ERP  products 
(e.g.,  system  releases)  is  also  responsible  for  ensuring  that  system  releases  are 
delivered  within  cost  and  schedule  constraints.  Because  performance  of  this 
system  development  and  management  role  makes  the  contractor  potentially 
unable  to  render  impartial  assistance  to  the  government  in  performing  the 
IV&V  function,  there  is  an  inherent  conflict  of  interest.  In  addition,  the  IV&V 
agent  reports  directly  and  solely  to  the  program  manager  and  not  to  program 
oversight  officials.  As  GAO  has  previously  reported,  the  IV&V  agent  should 
report  the  findings  and  associated  risks  to  program  oversight  officials,  as  well 
as  program  management,  in  order  to  better  ensure  that  the  IV&V  results  are 
objective  and  that  the  officials  responsible  for  making  program  investment 
decisions  are  fully  informed.  Furthermore,  the  contractor  has  largely  not 
produced  the  range  of  IV&V  deliverables  that  were  contractually  required 
between  2006  and  2008.  To  its  credit,  the  program  office  recently  began 
requiring  the  contractor  to  provide  assessment  reports,  as  required  under  the 
contract,  as  well  as  formal  quarterly  reports;  the  contractor  delivered  the 
results  of  the  first  planned  assessment  in  March  2009.  Notwithstanding  the 
recent  steps  that  the  program  office  has  taken,  it  nevertheless  lacks  an 
independent  perspective  on  the  program’s  products  and  management 
processes. 
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Abbreviations 

DOD 

Department  of  Defense 

DON 

Department  of  the  Navy 

ERP 

Enterprise  Resource  Planning 

FISC 

Fleet  Industrial  Supply  Center 

FOC 

full  operational  capability 

FOT&E 

follow-on  operational  test  and  evaluation 

GCSS-MC 

Global  Combat  Support  System — Marine  Corps 

GDIT 

General  Dynamics  Information  Technology 

IOC 

initial  operational  capability 

IOT&E 

initial  operational  test  and  evaluation 

1ST 

integrated  system  testing 

IT 

information  technology 

IV&V 

independent  verification  and  validation 

MDA 

milestone  decision  authority 

NAVAIR 

Naval  Air  Systems  Command 

NAVSEA 

Naval  Sea  Systems  Command 

NAVSUP 

Naval  Supply  Systems  Command 

NTCSS 

Naval  Tactical  Command  Support  System 

OT&E 

operational  test  and  evaluation 

SAP 

Systems  Applications  and  Products 

SPAWAR 

Space  and  Naval  Warfare  Systems  Command 

TEMP 

Test  and  Evaluation  Master  Plan 

UAT 

user  acceptance  testing 
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A  GAO 

^^Accountability  *  Integrity  *  Reliability 

United  States  Government  Accountability  Office 
Washington,  DC  20548 


September  15,  2009 

The  Honorable  Evan  Bayh 
Chairman 

The  Honorable  Richard  Burr 
Ranking  Member 

Subcommittee  on  Readiness  and  Management  Support 
Committee  on  Armed  Services 
United  States  Senate 

The  Honorable  John  Ensign 
United  States  Senate 

For  decades,  the  Department  of  the  Defense  (DOD)  has  been  challenged  in 
modernizing  its  timeworn  business  systems. 1  In  1995,  we  designated 
DOD’s  business  systems  modernization  program  as  high-risk,  and  continue 
to  do  so  today.2  Our  reasons  include  the  modernization’s  large  size, 
complexity,  and  its  critical  role  in  addressing  other  high-risk  areas,  such  as 
overall  business  transformation  and  financial  management.  Moreover,  we 
continue  to  report  on  business  system  investments  that  fail  to  effectively 
employ  acquisition  management  controls  and  deliver  promised  benefits 
and  capabilities  on  time  and  within  budget.3 

Nevertheless,  DOD  continues  to  invest  billions  of  dollars  in  thousands  of 
these  business  systems,  11  of  which  account  for  about  two-thirds  of  the 
department’s  annual  spending  on  business  programs.  The  Navy  Enterprise 
Resource  Planning  (ERP)  program  is  one  such  program.  Initiated  in  2003, 
Navy  ERP  is  to  standardize  the  Navy’s  acquisition,  financial,  program 
management,  plant  and  wholesale  supply,  and  workforce  management 


business  systems  are  information  systems,  including  financial  and  nonfinancial  systems 
that  support  DOD  business  operations,  such  as  civilian  personnel,  finance,  health,  logistics, 
military  personnel,  procurement,  and  transportation. 

2GAO,  High-Risk  Series:  An  Update,  GAO-09-271  (Washington,  D.C.:  January  2009). 

3See,  for  example,  GAO,  DOD  Business  Systems  Modernization:  Important  Management 
Controls  Being  Implemented  on  Major  Navy  Program,  but  Improvements  Needed  in  Key 
Areas,  GAO-08-896  (Washington,  D.C.:  Sept.  8,  2008);  DOD  Business  Systems 
Modernization:  Key  Marine  Co'tps  System  Acquisition  Needs  to  Be  Better  Justified, 
Defined,  and  Managed,  GAO-08-822  (Washington,  D.C.:  July  28,  2008);  and  DOD  Business 
Transformation:  Lack  of  an  Integrated  Strategy  Puts  the  Army's  Asset  Visibility  System 
Investments  at  Risk,  GAO-07-860  (Washington,  D.C.:  July  27,  2007). 
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business  processes  across  its  dispersed  organizational  environment.  As 
envisioned,  the  program  consists  of  a  series  of  major  increments,  the  first 
of  which  includes  three  releases  and  is  expected  to  cost  approximately 
$2.4  billion  over  its  20-year  life  cycle  and  to  be  fully  operational  in  fiscal 
year  2013.  We  recently  reported  that  Navy  ERP  program  management 
weaknesses  had  contributed  to  a  2-year  schedule  delay  and  about  $570 
million  in  cost  overruns.* * * 4 

As  agreed,  our  objectives  were  to  determine  whether  (1)  system  testing  is 
being  effectively  managed,  (2)  system  changes  are  being  effectively 
controlled,  and  (3)  independent  verification  and  validation  (IV&V) 
activities  are  being  effectively  managed.  To  accomplish  this,  we  analyzed 
relevant  program  documentation,  such  as  test  management  documents, 
individual  test  plans  and  procedures  and  related  test  results  and  defect 
reports;  system  change  procedures  and  specific  change  requests  and 
decisions;  change  review  board  minutes;  and  verification  and  validation 
plans  and  contract  documents.  We  also  observed  the  use  of  tools  for 
recording  and  tracking  test  defects  and  change  requests,  including  tracing 
a  statistically  valid  sample  of  transactions  through  these  tools. 

We  conducted  this  performance  audit  from  August  2008  to  September 
2009,  in  accordance  with  generally  accepted  government  auditing 
standards.  Those  standards  require  that  we  plan  and  perform  the  audit  to 
obtain  sufficient,  appropriate  evidence  to  provide  a  reasonable  basis  for 
our  findings  and  conclusions  based  on  our  audit  objectives.  We  believe 
that  the  evidence  obtained  provides  a  reasonable  basis  for  our  findings 
and  conclusions  based  on  our  audit  objectives.  Additional  details  on  our 
objectives,  scope,  and  methodology  are  in  appendix  I. 


Background 


The  Department  of  the  Navy’s  (DON)  primary  mission  is  to  organize,  train, 

maintain,  and  equip  combat-ready  naval  forces  capable  of  winning  wars, 
deterring  aggression  by  would-be  foes,  preserving  freedom  of  the  seas,  and 

promoting  peace  and  security.  Its  operating  forces,  known  as  the  fleet,  are 
supported  by  four  systems  commands.  Table  1  provides  a  brief  description 
of  each  command’s  responsibilities. 


4GAO-08-896. 


Page  2 


GAO-09-841  DOD  Business  Systems  Modernization 


Table  1:  Navy  Systems  Commands  and  Their  Responsibilities 

Systems  command 

Responsibilities 

Naval  Air  Systems  Command 
(NAVAIR) 

Developing,  delivering,  and  supporting  aircraft 
and  weapons  used  by  sailors  and  marines. 

Naval  Supply  Systems  Command 
(NAVSUP) 

Providing  supply,  fuel,  transportation,  and  other 
logistics  programs. 

Space  and  Naval  Warfare  Systems 
Command  (SPAWAR) 

Developing,  delivering,  and  supporting 
specialized  command  and  control  technologies, 
business  information  technology,  and  space 
capabilities. 

Naval  Sea  Systems  Command 
(NAVSEA) 

Acquiring  and  maintaining  the  department's 
ships  and  submarines. 

Source:  GAO  analysis  of  DON  data. 


To  support  the  department’s  mission,  these  commands  perform  a  variety 
of  interrelated  and  interdependent  business  functions  (e.g.,  acquisition  and 
financial  management),  relying  heavily  on  business  systems  to  do  so.  In 
fiscal  year  2009,  DON’s  budget  for  business  systems  and  associated 
infrastructure  was  about  $2.7  billion,  of  which  about  $2.2  billion  was 
allocated  to  operations  and  maintenance  of  existing  systems  and  about 
$500  million  to  systems  in  development  and  modernization.  Of  the 
approximately  2,480  business  systems  that  DOD  reports  having,  DON 
accounts  for  569,  or  about  23  percent,  of  the  total.  Navy  ERP  is  one  such 
system  investment. 


Navy  ERP:  A  Brief 
Description 


In  July  2003,  the  Assistant  Secretary  of  the  Navy  for  Research, 
Development,  and  Acquisition  established  Navy  ERP  to  converge  the 
functionality  of  four  pilot  systems  that  were  under  way  at  the  four 
commands  into  one  system.5  According  to  DOD,  Navy  ERP  is  to  address 
the  Navy’s  long-standing  problems  related  to  financial  transparency  and 
asset  visibility.  Specifically,  the  program  is  intended  to  standardize  the 
Navy’s  acquisition,  financial,  program  management,  plant  and  wholesale 
supply,  and  workforce  management  business  processes  across  its 
dispersed  organizational  components,  and  support  about  86,000  users 
when  fully  implemented. 

Navy  ERP  is  being  developed  in  a  series  of  increments  using  the  Systems 
Applications  and  Products  (SAP)  commercial  software  package, 


5The  four  pilots  are  SIGMA,  CABRILLO,  NEMAIS,  and  SMART. 
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augmented  as  needed  by  customized  software.  SAP  consists  of  multiple, 
integrated  functional  modules  that  perform  a  variety  of  business-related 
tasks,  such  as  finance  and  acquisition.  The  first  increment,  called 
Template  1,  is  currently  the  only  funded  portion  of  the  program  and 
consists  of  three  releases  (1.0,  1.1,  and  1.2). 6  Releasel.O,  Financial  and 
Acquisition,  is  the  largest  of  the  three  releases  in  terms  of  Template  1 
functional  requirements. 7  See  table  2  for  a  description  of  these  releases. 


Table  2:  Navy  ERP  Template  1  Releases 

Release 

Functionality 

1.0  Financial  and  Acquisition 

•  General  Fund  and  Navy  Working  Capital  Fund  finance  applications,  such  as  billing, 
budgeting,  and  cost  planning. 

•  Acquisition  applications,  such  as  activity-based  costing,  contract  awards,  and  budget  exhibits. 

•  Workforce  management  applications,  such  as  personnel  administration  and  training,  as  well 
as  events  management. 

1.1  Wholesale  and  Retail  Supply 

•  Wholesale  applications,  such  as  supply  and  demand  planning,  order  fulfillment,  and  supply 
forecasting. 

•  Retail  supply  applications,  such  as  inventory  management,  supply  and  demand  processing, 
and  warehouse  management. 

1.2  Intermediate-Level 
Maintenance 

•  Maintenance  applications,  such  as  maintenance  management,  quality  management,  and 
calibration  management. 

Source:  GAO  analysis  of  DON  data. 

DON  estimates  the  life-cycle  cost  for  Template  1  to  be  about  $2.4  billion, 
including  about  $1  billion  for  acquisition  and  $1.4  billion  for  operations 
and  maintenance.  The  program  office  reported  that  approximately  $600 
million  was  spent  from  fiscal  year  2004  through  fiscal  year  2008.  For  fiscal 
year  2009,  about  $190  million  is  planned  to  be  spent. 

To  acquire  and  deploy  Navy  ERP,  DON  established  a  program 
management  office  within  the  Program  Executive  Office  for  Executive 
Information  Systems.  The  program  office  manages  the  program’s  scope 
and  funding  and  is  responsible  for  ensuring  that  the  program  meets  its  key 
objectives.  To  accomplish  this,  the  program  office  performs  program 
management  functions,  including  testing,  change  control,  and  IV&V.  In 
addition,  various  DOD  and  DON  organizations  share  program  oversight 


6The  Navy  is  considering  deleting  the  third  release,  Release  1.2,  from  Template  1. 

'Release  1.0  accounts  for  about  56  percent  of  the  requirements;  Release  1.1,  about  33 
percent;  and  Release  1.2,  about  10  percent. 


Program  Oversight, 
Management,  and 
Contractor  Roles  and 
Responsibilities 
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and  review  activities.  A  listing  of  key  entities  and  their  roles  and 
responsibilities  is  provided  in  table  3. 

Table  3:  Organizations  Responsible  for  Navy  ERP  Oversight  and  Management 

Entity 

Roles  and  responsibilities 

Under  Secretary  of  Defense  for 
Acquisition,  Technology,  and  Logistics 

Serves  as  the  milestone  decision  authority  (M DA),  which  according  to  DOD,  has  overall 
responsibility  for  the  program,  to  include  approving  the  program  to  proceed  through  its 
acquisition  cycle  on  the  basis  of,  for  example,  independent  operational  test  evaluation  and 
certification. 

Assistant  Secretary  of  the  Navy, 

Research,  Development,  and  Acquisition 

Serves  as  DON'S  oversight  organization  forthe  program,  to  include  enforcement  of  Under 
Secretary  of  Defense  for  Acquisition,  Technology,  and  Logistics  policies  and  procedures. 

DON,  Program  Executive  Office  for 
Executive  Information  Systems 

Oversees  a  portfolio  of  large-scale  projects  and  programs  designed  to  enable  common 
business  processes  and  provide  standard  capabilities,  to  include  reviewing  and  approving 
overarching  test  plans  and  user  acceptance  test  readiness. 

Navy  ERP  Senior  Integration  Board 

Reviews  progress  in  attaining  acceptable  system  performance  at  systems  commands, 
including  approving  new  system  capabilities.  Chaired  by  the  Principal  Deputy  Assistant 
Secretary  of  the  Navy. 

Navy  ERP  Program  Management  Office 

Performs  day-to-day  program  management  and  serves  as  the  single  point  of 
accountability  for  managing  the  program's  objectives  through  development,  testing, 
deployment,  and  sustainment. 

Source:  GAO  analysis  of  DOD  data. 


To  deliver  system  and  other  program  capabilities  and  to  provide  program 
management  support  services,  Navy  ERP  relies  on  multiple  contractors,  as 
described  in  table  4. 


Table  4:  Navy  ERP  Program  Contracts 

Contract 

Award  date 

Completion 

date 

Contract  value 

Awarded  to 

Purpose 

Release  1.0  System 
Integration 

September 

2004 

February  2008 

$176  million 

BearingPoint 

Design  and  development  of 
release  1.0;  training  and 
deployment  at  NAVAIR 

Release  1.1  &  1.2  System 
Integration 

J  une  2007 

September 

2011 

$152.9  million 

IBM 

Design  and  development  of 
release  1.1  and  1.2 

Professional  Support 
Service  1 

J  une  2006 

September 

2010 

$163.7  million 

IBM 

Business  process  analysis, 
training,  organizational 
change  management,  and 
deployment  and  sustainment 
support 

Professional  Support 
Service  2 

J  une  2006 

September 

2010 

$69  million 

General  Dynamics 
Information  Technology 

S  upport  to  the  government  in 
its  oversight  of  the  system 

integrators  and  other 
contractors,  release 
management,  and  IV&V 


Source:  GAO  analysis  of  DON  data. 
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Overview  of  Navy  ERP’s 
Status 


Template  1  of  Navy  ERP  was  originally  planned  to  reach  full  operational 
capability  (FOC)  in  fiscal  year  2011,  and  its  original  estimated  life-cycle 
cost  was  about  $1.87  billion.8  The  estimate  was  later  baselined9  in  August 
2004  at  about  $2.0  billion.10  In  December  2006  and  again  in  September 
2007,  the  program  was  rebaselined.  FOC  is  now  planned  for  fiscal  year 
2013,  and  the  estimated  life-cycle  cost  is  about  $2.4  billion  (a  31  percent 
increase  over  the  original  estimate).11 

The  program  is  currently  in  the  production  and  deployment  phase  of  the 
defense  acquisition  system,  having  completed  the  system  development  and 
demonstration  phase  in  September  2007. 12  This  was  17  months  later  than 
the  program’s  original  schedule  set  in  August  2004,  but  on  time  according 
to  the  revised  schedule  set  in  December  2006.  Changes  in  the  program’s 
acquisition  phase  timeline  are  depicted  in  figure  1,  and  life-cycle  cost 
estimates  are  depicted  in  figure  2. 


8This  2003  estimate,  which  was  prepared  to  assist  in  budget  development  and  support  the 
Milestone  A/B  approval,  was  for  development,  deployment,  and  sustainment  costs  in  fiscal 
years  2003  through  2021. 

According  to  DOD’s  acquisition  guidebook,  an  Acquisition  Program  Baseline  is  a  program 
manager’s  estimated  cost,  schedule,  and  performance  goals.  Goals  consist  of  objective 
values,  which  represent  what  the  user  desires  and  expects,  and  threshold  values,  which 
represent  acceptable  limits.  When  the  program  manager  determines  that  a  current  cost, 
schedule,  or  performance  threshold  value  will  not  be  achieved,  the  MDA  must  be  notified, 
and  a  new  baseline  developed,  reviewed  by  decision  makers  and,  if  the  program  is  to 
continue,  approved  by  the  MDA. 

10 According  to  the  August  2004  Acquisition  Program  Baseline,  this  estimate  is  for 
acquisition,  operations,  and  support  for  fiscal  years  2004  through  2021. 

^According  to  the  September  2007  Acquisition  Program  Baseline,  this  estimate  is  for 
acquisition,  operations,  and  support  for  fiscal  years  2004  through  2023. 

12  The  defense  acquisition  system  is  a  framework-based  approach  that  is  intended  to 
translate  mission  needs  and  requirements  into  stable,  affordable,  and  well-managed 
acquisition  programs.  It  was  updated  in  December  2008  and  consists  of  five  key  program 
life-cycle  phases  and  three  related  milestone  decision  points — (1)  Materiel  Solution 
Analysis  (previously  Concept  Refinement),  followed  by  Milestone  A;  (2)  Technology 
Development,  followed  by  Milestone  B;  (3)  Engineering  and  Manufacturing  Development 
(previously  System  Development  and  Demonstration),  followed  by  Milestone  C;  (4) 
Production  and  Deployment;  and  (5)  Operations  and  Support. 
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Figure  1 :  Navy  ERP  Timeline 
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Source:  GAO  analysis  of  DON  data. 


Figure  2:  Navy  ERP  Life-Cycle  Cost  Estimates  in  Fiscal  Years  2003,  2004,  and  2007 

Dollars  in  billions 
3.0 


Fiscal  year 

Source:  GAO  analysis  of  DON  data. 
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Release  1.0  was  deployed  at  NAVAIR  in  October  2007,  after  passing 
developmental  testing  and  evaluation.  Initial  operational  capability  (IOC) 
was  achieved  in  May  2008,  22  months  later  than  the  baseline  established  in 
August  2004,  and  4  months  later  than  the  new  baseline  established  in 
September  2007.  According  to  program  documentation,  these  delays  were 
due,  in  part,  to  challenges  experienced  at  NAVAIR  in  converting  data  from 
legacy  systems  to  run  on  the  new  system  and  implementing  new  business 
procedures  associated  with  the  system.  In  light  of  the  delays  at  NAVAIR  in 
achieving  IOC,  the  deployment  schedules  for  the  other  commands  were 
revised  in  2008.  Release  1.0  was  deployed  at  NAVSUP  in  October  2008  as 
scheduled,  but  deployment  at  SPA  WAR  was  rescheduled  for  October  2009, 
18  months  later  than  planned,  and  at  NAVSEA  General  Fund  in  October 
2010,  and  at  Navy  Working  Capital  Fund  in  October  2011,  each  12  months 
later  than  planned. 

Release  1.1  is  currently  being  developed  and  tested,  and  is  planned  to  be 
deployed  at  NAVSUP  in  February  2010,  7  months  later  than  planned,  and 
at  the  Navy’s  Fleet  and  Industrial  Supply  Centers  (FISC)13  starting  in 
February  2011.  Changes  in  the  deployment  schedule  are  depicted  in 
figure  3. 


13F!eet  and  Industrial  Supply  Centers  are  located  in  San  Diego,  California;  Norfolk,  Virginia; 
Jacksonville,  Florida;  Puget  Sound,  Washington;  Pearl  Harbor,  Hawaii;  Yokosuka,  Japan; 
and  Sigonella,  Italy;  and  provide  worldwide  logistics  services  for  the  Navy. 
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Figure  3:  Navy  ERP  Deployment  Schedule 
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Source:  GAO  analysis  of  DON  data. 


Prior  GAO  Reviews  of 
DOD  Business  System 
Investments  Have 
Identified  IT  Management 
Weaknesses 


We  have  previously  reported  that  DOD  has  not  effectively  managed  key 
aspects  of  a  number  of  business  system  investments, 14  including  Navy 
ERP.  Among  other  things,  our  reviews  have  identified  weaknesses  in  such 
areas  as  architectural  alignment  and  informed  investment  decision 
making,  which  are  the  focus  of  the  Fiscal  Year  2005  Defense  Authorization 
Act  business  system  provisions. 15  Our  reviews  have  also  identified 
weaknesses  in  other  system  acquisition  and  investment  management 
areas,  such  as  earned  value  management, 16  economic  justification,  risk 


14See,  for  example,  GAO-08-896;  GAO,  DOD  Business  Systems  Modernization:  Planned 
Investment  in  Navy  Program  to  Create  Cashless  Shipboard  Environment  Needs  to  be 
Justified  and  Better  Managed,  GAO-08-922  (Washington,  D.C.:  Sept.  8,  2008);  GAO-08-822; 
GAO-07-860;  Information  Technology:  DOD  Needs  to  Ensure  that  Navy  Marine  Corps 
Intranet  Program  Is  Meeting  Goals  and  Satisfying  Customers,  GAO-07-51  (Washington, 
D.C.:  Dec.  8,  2006);  DOD  Systems  Modernization:  Planned  Investment  in  the  Navy 
Tactical  Command  Support  System.  Needs  to  be  Reassessed,  GAO-06-215  (Washington, 
D.C.:  Dec.  5,  2005);  and  DOD  Business  Systems  Modernization:  Navy  ERP  Adherence  to 
Best  Business  Practices  Critical  to  Avoid  Past  Failures,  GAO-05-858  (Washington,  D.C.: 
Sept.  29,  2005). 

15Ronald  W.  Reagan  National  Defense  Authorization  Act  for  Fiscal  Year  2005,  Pub.  L.  No. 
108-375,  Sec.  332  (2004)  (codified  at  10  U.S.C.  Sections  186  and  2222). 

16Eamed  value  management  is  a  means  for  measuring  actual  program  progress  against  cost 
and  schedule  estimates. 
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management,  requirements  management,  test  management,  and  IV&V 
practices. 

In  September  2008,  we  reported  that  DOD  had  implemented  key 
information  technology  (IT)  management  controls  on  Navy  ERP  to  varying 
degrees  of  effectiveness.17  For  example,  the  control  associated  with 
managing  system  requirements  had  been  effectively  implemented,  and 
important  aspects  of  other  controls  had  been  at  least  partially 
implemented,  including  those  associated  with  economically  justifying 
investment  in  the  program  and  proactively  managing  program  risks. 
However,  other  aspects  of  these  controls,  as  well  as  the  bulk  of  what  was 
needed  to  effectively  implement  earned  value  management,  had  not  been 
effectively  implemented.  As  a  result,  the  controls  that  were  not  effectively 
implemented  had,  in  part,  contributed  to  sizable  cost  and  schedule 
shortfalls.  Accordingly,  we  made  recommendations  aimed  at  improving 
cost  and  schedule  estimating,  earned  value  management,  and  risk 
management.  DOD  largely  agreed  with  our  recommendations. 

In  July  2008,  we  reported  that  DOD  had  not  implemented  key  aspects  of  its 
IT  acquisition  policies  and  related  guidance  on  its  Global  Combat  Support 
System-Marine  Corps  (GCSS-MC)  program.18  For  example,  we  reported 
that  it  had  not  economically  justified  its  investment  in  GCSS-MC  on  the 
basis  of  reliable  estimates  of  both  benefits  and  costs  and  had  not 
effectively  implemented  earned  value  management.  Moreover,  the 
program  office  had  not  adequately  managed  all  program  risks  and  had  not 
used  key  system  quality  measures.  We  concluded  that  by  not  effectively 
implementing  these  IT  management  controls,  the  program  was  at  risk  of 
not  delivering  a  system  solution  that  optimally  supports  corporate  mission 
needs,  maximizes  capability  mission  performance,  and  is  delivered  on  time 
and  within  budget.  Accordingly,  we  made  recommendations  aimed  at 
strengthening  cost  estimating,  schedule  estimating,  risk  management,  and 
system  quality  measurement.  The  department  largely  agreed  with  our 
recommendations. 

In  July  2007,  we  reported  that  the  Army’s  approach  for  investing  about  $5 
billion  in  three  related  programs — the  General  Fund  Enterprise  Business 
System,  Global  Combat  Support  System-Army  Field/Tactical,  and  Logistics 
Modernization  Program — did  not  include  alignment  with  the  Army 


17GAO-08-896. 

lsGAO-08-822. 
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enterprise  architecture  or  use  of  a  portfolio-based  business  system 
investment  review  process. 19  Further,  the  Logistics  Modernization 
Program’s  testing  was  not  adequate  and  had  contributed  to  the  Army’s 
inability  to  resolve  operational  problems.  In  addition,  the  Army  had  not 
established  an  IV&V  function  for  any  of  the  three  programs.  Accordingly, 
we  recommended,  among  other  things,  use  of  an  independent  test  team 
and  establishment  of  an  IV&V  function.  DOD  agreed  with  the 
recommendations. 

In  December  2005,  we  reported  that  DON  had  not,  among  other  things, 
economically  justified  its  ongoing  and  planned  investment  in  the  Naval 
Tactical  Command  Support  System  (NTCSS)  and  had  not  adequately 
conducted  requirements  management  and  testing  activities.20  Specifically, 
requirements  were  not  traceable  and  developmental  testing  had  not 
identified  problems  that,  subsequently,  twice  prevented  the  system  from 
passing  operational  testing.  Moreover,  DON  had  not  effectively  performed 
key  measurement,  reporting,  budgeting,  and  oversight  activities.  We 
concluded  that  DON  could  not  determine  whether  NTCSS,  as  defined  and 
as  being  developed,  was  the  right  solution  to  meet  its  strategic  business 
and  technological  needs.  Accordingly,  we  recommended  developing  the 
analytical  basis  necessary  to  know  if  continued  investment  in  NTCSS 
represented  a  prudent  use  of  limited  resources,  and  strengthening 
program  management,  conditional  upon  a  decision  to  proceed  with  further 
investment  in  the  program.  The  department  largely  agreed  with  our 
recommendations. 

In  September  2005,  we  reported  that  while  Navy  ERP  had  the  potential  to 
address  some  of  DON’s  financial  management  weaknesses,  it  faced 
significant  challenges  and  risks,  including  developing  and  implementing 
system  interfaces  with  other  systems  and  converting  data  from  legacy 
systems. 21  Also,  we  reported  that  the  program  was  not  capturing 
quantitative  data  to  assess  effectiveness,  and  had  not  established  an  IV&V 
function.  We  made  recommendations  to  address  these  areas,  including 
having  the  IV&V  agent  report  directly  to  program  oversight  bodies,  as  well 
as  the  program  manager.  DOD  generally  agreed  with  our 
recommendations,  including  that  an  IV&V  function  should  be  established. 


19GAO-07-860. 

zoGAO-06-215. 

zlGAO-05-858. 
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However,  it  stated  that  the  IV&V  team  would  report  directly  to  program 
management  who  in  turn  would  inform  program  oversight  officials  of  any 
significant  IV&V  results.  In  response,  we  reiterated  the  need  for  the  IV&V 
to  be  independent  of  the  program  and  stated  that  performing  IV&V 
activities  independently  of  the  development  and  management  functions 
helps  to  ensure  that  the  results  are  unbiased  and  based  on  objective 
evidence.  We  also  reiterated  our  support  for  the  recommendation  that  the 
IV&V  reports  be  provided  to  the  appropriate  oversight  body  so  that  it  can 
determine  whether  any  of  the  IV&V  results  are  significant.  We  noted  that 
doing  so  would  give  added  assurance  that  the  results  were  objective  and 
that  those  responsible  for  authorizing  future  investments  in  Navy  ERP 
have  the  information  needed  to  make  informed  decisions. 


Key  Aspects  of  Navy 
ERP  Testing  Have 
Been  Effectively 
Managed 


To  be  effectively  managed,  testing  should  be  planned  and  conducted  in  a 
structured  and  disciplined  fashion.  According  to  DOD  and  industry 
guidance,22  system  testing  should  be  progressive,  meaning  that  it  should 
consist  of  a  series  of  test  events  that  first  focus  on  the  performance  of 
individual  system  components,  then  on  the  performance  of  integrated 
system  components,  followed  by  system-level  tests  that  focus  on  whether 
the  entire  system  (or  major  system  increments)  is  acceptable, 
interoperable  with  related  systems,  and  operationally  suitable  to  users. 
For  this  series  of  related  test  events  to  be  conducted  effectively,  all  test 
events  need  to  be,  among  other  things,  governed  by  a  well-defined  test 
management  structure  and  adequately  planned.  Further,  the  results  of 
each  test  event  need  to  be  captured  and  used  to  ensure  that  problems 
discovered  are  disclosed  and  corrected. 


Key  aspects  of  Navy  ERP  testing  have  been  effectively  managed. 
Specifically,  the  program  has  established  an  effective  test  management 
structure,  key  development  events  were  based  on  well-defined  plans,  the 
results  of  all  executed  test  events  were  documented,  and  problems  found 
during  testing  (i.e.,  test  defects)  were  captured  in  a  test  management  tool 
and  subsequently  analyzed,  resolved,  and  disclosed  to  decision  makers. 


Z2See,  for  example,  Office  of  the  Under  Secretary  of  Defense  for  Acquisition,  Technology, 
and  Logistics,  Department  of  Defense  Instruction  5000.02  (Arlington,  VA:  Dec.  2,  2008); 
Defense  Acquisition  University,  Test  and  Evaluation  Management  Guide,  5th  ed.  (Fort 
Belvoir,  VA:  January  2005);  Institute  of  Electrical  and  Electronics  Engineers,  Inc.,  Standard 
for  Software  Verification  and  Validation,  IEEE  Std  1012-2004  (New  York,  NY:  June  8, 
2005);  Software  Engineering  Institute,  Capability  Maturity  Model  Integration  for 
Acquisition,  version  1.2  (Pittsburgh,  PA:  May  2008);  and  GAO,  Year  2000  Computing 
Crisis:  A  Testing  Guide,  GAO/AIMD-IO.1.21  (Washington,  D.C.:  November  1998). 
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Further,  while  we  identified  instances  in  which  the  tool  did  not  contain 
key  data  about  defects  that  are  needed  to  ensure  that  unauthorized 
changes  to  the  status  of  defects  do  not  occur,  the  number  of  instances 
found  are  not  sufficient  to  conclude  that  the  controls  were  not  operating 
effectively.  Notwithstanding  the  missing  data,  this  means  that  Navy  ERP 
testing  has  been  performed  in  a  manner  that  increases  the  chances  that  the 
system  will  meet  operational  needs  and  perform  as  intended. 


A  Well-defined  Test  The  program  office  has  established  a  test  management  structure  that 

Management  Structure  Has  satisfies  key  elements  of  DOD  and  industry  guidance.23  For  example,  the 
Been  Established  program  has  developed  a  Test  and  Evaluation  Master  Plan  (TEMP)  that 

defines  the  program’s  test  strategy.  As  provided  for  in  the  guidance,  this 
strategy  consists  of  a  sequence  of  tests  in  a  simulated  environment  to 
verify  first  that  individual  system  parts  meet  specified  requirements  (i.e., 
development  testing)  and  then  verify  that  these  combined  parts  perform  as 
intended  in  an  operational  environment  (i.e.,  operational  testing).  As  we 
have  previously  reported,24  such  a  sequencing  of  test  events  is  an  effective 
approach  because  it  permits  the  source  of  defects  to  be  isolated  sooner, 
before  it  is  more  difficult  and  expensive  to  address. 

More  specifically,  the  strategy  includes  a  sequence  of  developmental  tests 
for  each  release  consisting  of  three  cycles  of  integrated  system  testing 
(1ST)  followed  by  user  acceptance  testing  (UAT).  Following  development 
testing,  the  sequence  of  operational  tests  includes  the  Navy’s  independent 
operational  test  agency  conducting  initial  operational  test  and  evaluation 
(IOT&E)  and  then  follow-on  operational  test  and  evaluation  (FOT&E),  as 
needed,  to  validate  the  resolution  of  deficiencies  found  during  IOT&E.  See 
table  5  for  a  brief  description  of  the  purpose  of  each  test  activity,  and 
figure  4  for  the  schedule  of  Release  1.0  and  1.1  test  activities. 


"3Office  of  the  Under  Secretary  of  Defense  for  Acquisition,  Technology,  and  Logistics, 
Department  of  Defense  Instruction  5000.02  (Arlington,  VA:  Dec.  2,  2008);  Defense 
Acquisition  University,  Test  and  Evaluation  Management  Guide,  5th  ed.  (Fort  Belvoir,  VA: 
January  2005);  and  Institute  of  Electrical  and  Electronics  Engineers,  Inc.,  Standard  for 
Software  and  System,  Test  Documentation,  IEEE  Std  829-2008  (New  York,  NY:  2008). 

24GAO,  Secure  Border  Initiative:  DHS  Needs  to  Address  Significant  Risks  in  Delive?i?ig 
Key  Technology  Investment,,  GAO-08-1086  (Washington,  D.C.:  Sept.  22,  2008). 


Page  13 


GAO-09-841  DOD  Business  Systems  Modernization 


Table  5:  Description  of  the  Purpose  of  Navy  ERP  Tests 

Test 

Purpose 

Developmental  testing 

1ST 

To  validate  that  the  technical  and  functional  components  of  the  system  work 
properly  together  and  operate  as  specified  by  the  requirements. 

Cycle  1  (Scenario  Testing) 

To  validate  chains  of  business  process  transactions  using  small  scenarios,  such 
as  a  standard  sales  order,  delivery,  and  invoicing.  Also,  independent  evaluators 
observe  scenario  testing  in  preparation  for  operational  test  and  evaluation. 

Cycle  2  (Scenario  Testing  and  Conversions  and 
Interfaces) 

To  validate  more  complex  sequences  of  transactions  plus  customized  software. 

Cycle  3  (Final  Integration  Testing) 

To  validate  the  entire  system,  including  external  components. 

UAT 

To  allow  the  customerto  ensure  Navy  ERP  works  properly  and  operates  as 
specified  by  the  requirements. 

Operational  testing 

IOT&E 

To  evaluate  the  operational  effectiveness  and  suitability  of  the  system. 

FOT&E 

To  verify  the  correction  of  deficiencies  identified  during  IOT&E. 

Source: 

GAO  analysis  of  DON  data. 

Figure  4:  Release  1.0  and  1.1  Test  Activity  Schedule 
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The  TEMP  also  clearly  identifies  the  roles  and  responsibilities  of  key  Navy 
ERP  testing  organizations,  as  provided  for  in  DOD  and  industry  guidance. 
For  example,  it  describes  specific  responsibilities  of  the  program  manager, 
system  integrator,  quality  assurance/test  team  lead,  and  independent 
operational  test  and  evaluation  organizations.  Table  6  summarizes  the 
responsibilities  of  these  various  test  organizations. 


Table  6:  Navy  ERP  Testing-Related  Organizations  and  Respective  Roles  and  Responsibilities 

Testing-related  organization 

Responsibilities 

Program  manager 

Provides  overall  management  and  direction  of  Navy  ERP  testand  evaluation. 

Conducts  test  readiness  reviews. 

Certifies  that  the  program  is  ready  to  proceed  from  developmental  to  operational  testing 
in  a  developmental  testand  evaluation  report. 

System  integrator 

S  upports  the  execution  of  integration  and  user  acceptance  testing,  including  training 
system  testers  and  users. 

Reports  to  the  Navy  ERP  program  manager. 

Quality  assurance/test  team  lead 

Creates  the  test  and  evaluation  strategy  and  developmental  test  and  evaluation  plan. 

Assists  in  planning,  coordinating,  and  conducting  developmental  testing  and  evaluation, 
and  reporting  the  results  to  the  program  manager. 

Conducts  integration  testing. 

Operational  Testand  Evaluation  Force 

Plans  and  conducts  Navy  ERP  operational  testand  evaluation  (OT&E). 

Reports  results  and  recommendations  to  DOD's  Director,  Operational  Testand 

Evaluation. 

Performs  follow-on  OT&E  to  verify  that  deficiencies  found  during  initial  OT&E  have  been 
resolved. 

J  oint  Interoperability  Test  Command 

Certifies  to  the  J  ointChiefs  of  Staff  that  interoperability  requirements  are  met. 

Verifies  readiness  for  interoperability  to  the  responsible  operational  test  agency  during  or 
prior  to  operational  test  readiness  review. 

Office  of  Director,  Operational  Test  and 
Evaluation 

Reviews  and  approves  IOT&E  and  FOT&E  plans. 

Analyzes  OT&E  results. 

Provides  independent  assessment  to  the  MDA. 

Source:  GAO  analysis  of  DOD  data. 

Well-defined  Plans  for 
Developmental  Test  Events 
Were  Developed 

According  to  relevant  guidance,25  test  activities  should  be  governed  by 
well-defined  and  approved  plans.  Among  other  things,  such  plans  are  to 
include  a  defect  triage  process,  metrics  for  measuring  progress  in 
resolving  defects,  test  entrance  and  exit  criteria,  and  test  readiness 
reviews. 

2BSee,  for  example,  GAO/AIMD-IO.1.21. 
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Each  developmental  test  event  for  Release  1.0  (i.e.,  each  cycle  of 
integrated  systems  testing  and  user  acceptance  testing)  was  based  on  a 
well-defined  test  plan.  For  example,  each  plan  provided  for  conducting 
daily  triage  meetings  to  (1)  assign  new  defects  a  criticality  level  using 
documented  criteria,26  (2)  record  new  defects  and  update  the  status  of  old 
defects  in  the  test  management  tool,  and  (3)  address  other  defect  and 
testing  issues.  Further,  each  plan  included  defect  metrics,  such  as  the 
number  of  defects  found  and  corrected  and  their  age.  In  addition,  each 
plan  specified  that  testing  was  not  complete  until  all  major  defects  found 
during  the  cycle  were  resolved,  and  all  unresolved  defects’  impact  on  the 
next  test  event  were  understood.  Further,  the  plans  provided  for  holding 
test  readiness  reviews  to  review  test  results  as  a  condition  for  proceeding 
to  the  next  event.  By  ensuring  that  plans  for  key  development  test 
activities  include  these  aspects  of  effective  test  planning,  the  risk  of  test 
activities  not  being  effectively  and  efficiently  performed  is  reduced,  thus 
increasing  the  chances  that  the  system  will  meet  operational  requirements 
and  perform  as  intended. 


Test  Results  Were 
Documented  and 
Reported,  but  Key 
Information  about 
Changes  to  the  Status  of 
Reported  Defects  Was  Not 
Always  Recorded 


According  to  industry  guidance,27  effective  system  testing  includes 
capturing,  analyzing,  resolving,  and  disclosing  to  decision  makers  the 
status  of  problems  found  during  testing  (i.e.,  test  defects).  Further,  this 
guidance  states  that  these  results  should  be  collected  and  stored  according 
to  defined  procedures  and  placed  under  appropriate  levels  of  control  to 
ensure  that  any  changes  to  the  results  are  fully  documented. 

To  the  program’s  credit,  the  relevant  testing  organizations  have 
documented  test  defects  in  accordance  with  defined  plans.  For  example, 
daily  triage  meetings  involving  the  test  team  lead,  testers,  and  functional 
experts  were  held  to  review  each  new  defect,  assign  it  a  criticality  level, 
and  designate  someone  responsible  for  resolving  it  and  for  monitoring  and 


26 According  to  program  documentation,  criticality  levels  range  from  1  to  5,  as  follows:  1  is  a 
problem  that  prevents  accomplishment  of  an  operational  or  mission  critical  capability;  2  is 
a  major  technical  problem  with  no  work-around  solution;  3  is  a  major  technical  problem 
with  a  work-around  solution;  4  is  a  minor  technical  problem;  and  5  is  any  other  defect,  such 
as  a  cosmetic  problem. 

"‘Institute  of  Electrical  and  Electronics  Engineers,  Inc.,  Standard  for  Information 
Technology — Software  Life  Cycle  Processes — Implementation  Considerations,  IEEE/EIA 
Std  12207.2-1997  (New  York,  NY:  April  1998)  and  Software  Engineering  Institute, 

Capability  Maturity  Model  Integration  for  Acquisition,  version  1.2  (Pittsburgh,  PA:  May 
2008). 
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updating  its  resolution  in  the  test  management  tool.  Further,  test  readiness 
reviews  were  conducted  at  which  entrance  and  exit  criteria  for  each  key 
test  event  were  evaluated  before  proceeding  to  the  next  event.  As  part  of 
these  reviews,  the  program  office  and  oversight  officials,  command 
representatives,  and  test  officials  reviewed  the  results  of  test  events  to 
ensure,  among  other  things,  that  significant  defects  were  closed  and  that 
there  were  no  unresolved  defects  that  could  affect  execution  of  the  next 
test  event. 

However,  the  test  management  tool  did  not  always  contain  key  data  for  all 
recorded  defects  that  are  needed  to  ensure  that  unauthorized  changes  to 
the  status  of  defects  do  not  occur.  According  to  information  systems 
auditing  guidelines,28  audit  tools  should  be  in  place  to  monitor  user  access 
to  systems  to  detect  possible  errors  or  unauthorized  changes.  For  Navy 
ERP,  this  was  not  always  the  case.  Specifically,  while  the  tool  has  the 
capability  to  track  changes  to  test  defects  in  a  history  log,29  our  analysis  of 
80  randomly  selected  defects  in  the  tool  disclosed  two  instances  in  which 
the  tool  did  not  record  when  a  change  in  the  defect’s  status  was  made  or 
who  made  the  change.  In  addition,  our  analysis  of  12  additional  defects 
that  were  potential  anomalies30  disclosed  two  additional  instances  where 
the  tool  did  not  record  when  a  change  was  made  and  who  made  it.  While 
our  sample  size  and  results  do  not  support  any  conclusions  as  to  the 
overall  effectiveness  of  the  controls  in  place  for  recording  and  tracking 
test  defect  status  changes,  they  do  show  that  it  is  possible  that  changes 
can  be  made  without  a  complete  audit  trail  surrounding  those  changes. 
After  we  shared  our  results  with  program  officials,  they  stated  that  they 
provided  each  instance  for  resolution  to  the  vendor  responsible  for  the 
tracking  tool.  These  officials  attributed  these  instances  to  vendor  updates 
to  the  tool  that  caused  the  history  settings  to  default  to  “off.”  To  address 


^Information  Systems  Audit  and  Control  Association,  Inc.,  IS  Standards,  Guidelines  and 
Procedures  for  Auditing  and  Control  Professionals  (Rolling  Meadows,  IL:  Jan.  15,  2009). 

29 According  to  program  documentation,  the  date  a  defect  is  entered  into  the  system  and  the 
date  the  status  of  the  defect  is  changed  to  “closed”  are  automatically  populated.  Further, 
changes  to  a  defect’s  status,  including  from  “new”  to  “open”  and  from  “open”  to  “closed;” 
changes  to  the  criticality  level;  and  the  user  who  makes  the  changes  are  tracked  in  a 
defect’s  history  log. 

30These  anomalies  are  defects  that  we  found  that  (1)  were  attributed  to  integrated  system 
test  events,  but  were  not  detected  until  after  the  system  was  deployed;  (2)  had  a  criticality 
level  that  was  different  from  the  level  that  was  reported  at  a  test  readiness  review;  (3)  were 
deferred  to  a  later  test  event  or  to  post-deployment  to  be  verified  as  resolved;  or  (4)  had  no 
criticality  level. 
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this  weakness,  they  added  that  they  are  now  ensuring  that  the  history  logs 
are  set  correctly  after  any  update  to  the  tool.  This  addition  is  a  positive 
step  because  without  an  effective  information  system  access  audit  tool, 
the  probability  of  test  defect  status  errors  or  unauthorized  changes  is 
increased. 


System  Changes  Have 
Been  Controlled,  but 
Their  Cost  and 
Schedule  Impacts 
Were  Not  Sufficiently 
Considered 


Industry  best  practices  and  DOD  guidance31  recognize  the  importance  of 
system  change  control  when  developing  and  maintaining  a  system.  Once 
the  composition  of  a  system  is  sufficiently  defined,  a  baseline 
configuration  is  normally  established,  and  changes  to  that  baseline  are 
placed  under  a  disciplined  change  control  process  to  ensure  that 
unjustified  and  unauthorized  changes  are  not  introduced.  Elements  of 
disciplined  change  control  include  (1)  formally  documenting  a  change 
control  process,  (2)  rigorously  adhering  to  the  documented  process,  and 
(3)  adopting  objective  criteria  for  considering  a  proposed  change, 
including  its  estimated  cost  and  schedule  impact. 


To  its  credit,  the  Navy  ERP  program  has  formally  documented  a  change 
control  process.  Specifically,  it  has  a  plan  and  related  procedures  that 
include  the  purpose  and  scope  of  the  process — to  ensure  that  any  changes 
made  to  the  system  are  properly  identified,  developed,  and  implemented  in 
a  defined  and  controlled  environment.  It  also  is  using  an  automated  tool  to 
capture  and  track  the  disposition  of  each  change  request.  Further,  it  has 
defined  roles  and  responsibilities  and  a  related  decision-making  structure 
for  reviewing  and  approving  system  changes.  In  this  regard,  the  program 
has  established  a  hierarchy  of  review  and  approval  boards,  including  a 
Configuration  Control  Board  to  review  all  changes  and  a  Configuration 
Management  Board  to  further  review  changes  estimated  to  require  more 
than  100  hours  or  $25,000  to  implement.  Furthermore,  a  Navy  ERP  Senior 
Integration  Board  was  recently  established  to  review  and  approve  requests 
to  add,  delete,  or  change  the  program’s  requirements.  In  addition,  the 
change  control  process  states  that  the  decisions  are  to  be  based  on,  among 
others,  the  system  engineering  and  earned  value  management  (i.e.,  cost 
and  schedule)  impacts  the  change  will  introduce,  such  as  the  estimated 
number  of  work  hours  that  will  be  required  to  effect  the  change.  Table  7 


31See,  for  example,  Electronics  Industries  Alliance,  National  Consensus  Standard  for 
Configuration  Management ,  ANSI/EIA-649-1998  (Arlington,  VA:  August  1998)  and 
Department  of  Defense,  Military  Handbook:  Configuration  Management  Guidance,  MIL- 
HDBK-61A(SE)  (Washington,  D.C.:  Feb.  7,  2001). 
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provides  a  brief  description  of  the  decision-making  authorities  and  boards 
and  their  respective  roles  and  responsibilities. 

Table  7:  Roles  and  Responsibilities  for  Change  Review  and  Approval 

Review  and  approval  organizations 

Roles  and  responsibilities 

Navy  ERP  Senior  Integration  Board 

Reviews  and  approves  Engineering  Change  Proposals,  which  are  proposed  changes  that 
would  impact  system  scope,  configuration,  cost,  or  schedule  by  adding,  deleting,  or 
changing  requirements.  The  board  is  chaired  by  the  Principal  Deputy  Assistant  Secretary  of 
the  Navy,  Research,  Development,  and  Acquisition. 

Configuration  Management  Board 

Reviews  and  approves  change  requests  requiring  more  than  100  hours  or  $25,000  to 
implement.  The  board  is  chaired  by  the  program  manager  and  includes  representatives  from 
the  earned  value  management  team  (i.e.,  costand  schedule). 

Configuration  Control  Board 

Reviews  all  change  requests  and  approves  those  requiring  less  than  100  hours  or  $25,000 
to  implement.  The  board  is  chaired  by  the  systems  engineer  and  includes  representatives 
from  the  earned  value  management  team  (i.e.,  costand  schedule). 

Engineering  Review  Board 

Ensures  change  requests  are  ready  to  proceed  to  the  Configuration  Control  Board  by 
reviewing  and  recommending  changes.  This  board  is  facilitated  and  chaired  by  the  systems 
engineer  and  the  configuration  manager  to  ensure  the  change  request  documentation  is 
complete. 

Technical  Change  Control  Board 

Approves  or  defers  transport  change  requests,  which  are  requests  to  release  changes  into 
the  deployed  system.  The  board  is  chaired  by  the  production  manager. 

Source:  GAO  analysis  of  DON  documentation. 


Navy  ERP  is  largely  adhering  to  its  documented  change  control  process. 
Specifically,  our  review  of  a  random  sample  of  60  change  requests  and 
minutes  of  related  board  meetings  held  between  May  2006  and  April  2009 
showed  that  the  change  requests  were  captured  and  tracked  using  an 
automated  tool,  and  they  were  reviewed  and  approved  by  the  designated 
decision-making  authorities  and  boards,  in  accordance  with  the  program’s 
documented  process. 

However,  the  program  has  not  sufficiently  or  consistently  considered  the 
cost  and  schedule  impacts  of  proposed  changes.  Our  analysis  of  the 
random  sample  of  60  change  requests,  including  our  review  of  related 
board  meeting  minutes,  showed  no  evidence  that  cost  and  schedule 
impacts  were  identified  or  that  they  were  considered.  Specifically,  we  did 
not  see  evidence  that  the  cost  and  schedule  impacts  of  these  change 
requests  were  assessed.  According  to  program  officials,  the  cost  and 
schedule  impacts  of  each  change  were  discussed  at  control  board 
meetings.  In  addition,  they  provided  two  change  requests  to  demonstrate 
this.  However,  while  these  change  requests  did  include  schedule  impact, 
they  did  not  include  the  anticipated  cost  impact  of  proposed  changes. 
Rather,  these  two,  as  well  as  those  in  our  random  sample,  included  the 
estimated  number  of  work  hours  required  to  implement  the  change. 
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Because  the  cost  of  any  proposed  change  depends  on  other  factors 
besides  work  hours,  such  as  labor  rates,  the  estimated  number  of  work 
hours  is  not  sufficient  for  considering  the  cost  impact  of  a  change.  In  the 
absence  of  verifiable  evidence  that  cost  and  schedule  impacts  were 
consistently  considered,  approval  authorities  do  not  appear  to  have  been 
provided  key  information  needed  to  fully  inform  their  decisions  on 
whether  or  not  to  approve  a  change.  System  changes  that  are  approved 
without  a  full  understanding  of  their  cost  and  schedule  impacts  could 
result  in  unwarranted  cost  increases  and  schedule  delays. 


Navy  ERP  IV&V 
Function  Is  Not 
Independent  and 
Not  Been  Fully 
Performed 


Has 


The  purpose  of  IV&V  is  to  independently  ensure  that  program  processes 
and  products  meet  quality  standards.  The  use  of  an  IV&V  function  is 
recognized  as  an  effective  practice  for  large  and  complex  system 
development  and  acquisition  programs,  like  Navy  ERP,  as  it  provides 
objective  insight  into  the  program’s  processes  and  associated  work 
products.32  To  be  effective,  verification  and  validation  activities  should  be 
performed  by  an  entity  that  is  managerially  independent  of  the  system 
development  and  management  processes  and  products  that  are  being 
reviewed.33  Among  other  things,  such  independence  helps  to  ensure  that 
the  results  are  unbiased  and  based  on  objective  evidence. 


The  Navy  has  not  effectively  managed  its  IV&V  function  because  it  has  not 
ensured  that  the  contractor  performing  this  function  is  independent  of  the 
products  and  processes  that  this  contractor  is  reviewing  and  because  it 
has  not  ensured  that  the  contractor  is  meeting  contractual  requirements. 
In  June  2006,  DON  awarded  a  professional  support  services  contract  to 
General  Dynamics  Information  Technology  (GDIT),  to  include 
responsibilities  for,  among  other  things,  IV&V,  program  management 
support,  and  delivery  of  releases  according  to  cost  and  schedule 
constraints.  According  to  the  program  manager,  the  contractor’s  IV&V 
function  is  organizationally  separate  from,  and  thus  independent  of,  the 
contractor’s  Navy  ERP  system  development  function.  However,  the 
subcontractor  performing  the  IV&V  function  is  also  performing  release 


32GAO,  Homeland  Security:  U.S.  Visitor  and  Immigration  Status  Indicator  Technology 
Program  Planning  and  Execution  Improvements  Needed,  GAO-09-96  (Washington,  D.C.: 
Dec.  12,  2008);  Homeland  Security:  Recommendations  to  Improve  Management  of  Key 
Border  Security  Program  Need  to  Be  Implemented,  GAO-06-296  (Washington,  D.C.:  Feb. 

14,  2006);  and  GAO-05-858. 

33Institute  of  Electrical  and  Electronics  Engineers,  Inc.,  Standard  for  Software  Verification 
and.  Validation,  IEEE  Std  1012-2004  (New  York,  NY:  June  8,  2005). 
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management.  According  to  the  GDIT  contract,  the  release  manager  is 
responsible  for  developing  and  deploying  a  system  release  that  meets 
operational  requirements  within  the  program’s  cost  and  schedule 
constraints,  but  it  also  states  that  the  IV&V  function  is  responsible  for 
supporting  the  government  in  its  review,  approval,  and  acceptance  of  Navy 
ERP  products  (e.g.,  releases).  The  contract  also  states  that  GDIT  is  eligible 
for  an  optional  award  fee  payment  based  on  its  performance  in  meeting, 
among  other  things,  these  cost  and  schedule  constraints.  Because 
performance  of  the  system  development  and  management  role  makes  the 
contractor  potentially  unable  to  render  impartial  assistance  to  the 
government  in  performing  the  IV&V  function,  the  contractor  has  an 
inherent  conflict  of  interest  relative  to  meeting  cost  and  schedule 
commitments  and  disclosing  the  results  of  verification  and  validation 
reviews  that  may  affect  its  ability  to  do  so. 

The  IV&V  function’s  lack  of  independence  is  amplified  by  the  fact  that  it 
reports  directly  and  solely  to  the  program  manager.  As  we  have  previously 
reported,34  the  IV&V  function  should  report  the  issues  or  weaknesses  that 
increase  the  risks  associated  with  the  project  to  program  oversight 
officials,  as  well  as  to  program  management,  to  better  ensure  that  the 
verification  and  validation  results  are  objective  and  that  the  officials 
responsible  for  making  program  investment  decisions  are  fully  informed. 
Furthermore,  these  officials,  once  informed,  can  ensure  that  the  issues  or 
weaknesses  reported  are  promptly  addressed. 

Without  ensuring  sufficient  managerial  independence,  valuable 
information  may  not  reach  decision  makers,  potentially  leading  to  the 
release  of  a  system  that  does  not  adequately  meet  users’  needs  and  operate 
as  intended. 

Beyond  the  IV&V  function’s  lack  of  independence,  the  program  office  has 
not  ensured  that  the  subcontractor  has  produced  the  range  of  deliverables 
that  were  contractually  required  and  defined  in  the  IV&V  plan.  For 
example,  the  contract  and  plan  call  for  weekly  and  monthly  reports 
identifying  weaknesses  in  program  processes  and  recommendations  for 
improvement,  a  work  plan  for  accomplishing  IV&V  tasks,  and  associated 
assessment  reports  that  follow  the  System  Engineering  Plan  and  program 
schedule.  However,  the  IV&V  contractor  has  largely  not  delivered  these 
products.  Specifically,  until  recently,  it  did  not  produce  a  work  plan  and 


34See  GAO-07-860  and  GAO-05-858. 
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only  monthly  reports  were  delivered,  and  these  reports  only  list  meetings 
that  the  IV&V  contactor  attended  and  documents  that  it  reviewed.  They  do 
not,  for  example,  identify  program  weaknesses  or  provide 
recommendations  for  improvement.  According  to  program  officials,  they 
have  relied  on  oral  reports  from  the  subcontractor  at  weekly  meetings, 
and  these  lessons  learned  have  been  incorporated  into  program  guidance. 
According  to  the  contractor,  the  Navy  has  expended  about  $1.8  million 
between  June  2006  and  September  2008  for  IV&V  activities,  with  an 
additional  $249,000  planned  to  be  spent  in  fiscal  year  2009. 

Following  our  inquiries  about  an  IV&V  work  plan,  the  IV&V  contractor 
developed  such  a  plan  in  October  2008,  more  than  2  years  after  the 
contract  was  awarded,  that  lists  program  activities  and  processes  to  be 
assessed,  such  as  configuration  management  and  testing.  While  this  plan 
does  not  include  time  frames  for  starting  and  completing  these 
assessments,  meeting  minutes  show  that  the  status  of  assessments  has 
been  discussed  with  the  program  manager  during  IV&V  review  meetings. 
The  first  planned  assessment  was  delivered  to  the  program  in  March  2009 
and  provides  recommendations  for  improving  the  program’s  configuration 
management  process,  such  as  using  the  automated  tool  to  produce  certain 
reports  and  enhancing  training  to  understand  how  the  tool  is  used. 

Further,  program  officials  stated  that  they  have  also  recently  begun 
requiring  the  contractor  to  provide  formal  quarterly  reports,  the  first  of 
which  was  delivered  to  the  program  manager  in  January  2009.  Our  review 
of  this  quarterly  report  shows  that  it  provides  recommendations  for 
improving  the  program’s  risk  management  process  and  organizational 
change  management  strategy. 

Notwithstanding  the  recent  steps  that  the  program  office  has  taken,  it 
nevertheless  lacks  an  independent  perspective  on  the  program’s  products 
and  management  processes. 


Conclusions 


DOD’s  successes  in  delivering  large-scale  business  systems,  such  as  Navy 
ERP,  are  in  large  part  determined  by  the  extent  to  which  it  employs  the 
kind  of  rigorous  and  disciplined  IT  management  controls  that  are  reflected 
in  department  policies  and  related  guidance.  While  implementing  these 
controls  does  not  guarantee  a  successful  program,  it  does  minimize  a 
program’s  exposure  to  risk  and  thus  the  likelihood  that  it  will  fall  short  of 
expectations.  In  the  case  of  Navy  ERP,  living  up  to  expectations  is 
important  because  the  program  is  large,  complex,  and  critical  to 
addressing  the  department’s  long-standing  problems  related  to  financial 
transparency  and  asset  visibility. 
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The  Navy  ERP  program  office  has  largely  implemented  a  range  of  effective 
controls  associated  with  system  testing  and  change  control,  including 
acting  quickly  to  address  issues  with  the  audit  log  for  its  test  management 
tool,  but  more  can  be  done  to  ensure  that  the  cost  and  schedule  impacts  of 
proposed  changes  are  explicitly  documented  and  considered  when 
decisions  are  reached.  Moreover,  while  the  program  office  has  contracted 
for  IV&V  activities,  it  has  not  ensured  that  the  contractor  is  independent  of 
the  products  and  processes  that  it  is  to  review  and  has  not  held  the 
contractor  accountable  for  producing  the  full  range  of  IV&V  deliverables 
required  under  the  contract.  Moreover,  it  has  not  ensured  that  its  IV&V 
contractor  is  accountable  to  a  level  of  management  above  the  program 
office,  as  we  previously  recommended.  Notwithstanding  the  program 
office’s  considerable  effectiveness  in  how  it  has  managed  both  system 
testing  and  change  control,  these  weaknesses  increase  the  risk  of  investing 
in  system  changes  that  are  not  economically  justified  and  unnecessarily 
limit  the  value  that  an  IV&V  agent  can  bring  to  a  program  like  Navy  ERP. 
By  addressing  these  weaknesses,  the  department  can  better  ensure  that 
taxpayer  dollars  are  wisely  and  prudently  invested. 


RppnmmPnrlBtinns  for  Strengthen  the  management  of  Navy  ERP’s  change  control  process,  we 

recommend  that  the  Secretary  of  Defense  direct  the  Secretary  of  the  Navy, 
Executive  Action  through  the  appropriate  chain  of  command,  to  (1)  revise  the  Navy  ERP 

procedures  for  controlling  system  changes  to  explicitly  require  that  a 
proposed  change’s  life-cycle  cost  impact  be  estimated  and  considered  in 
making  change  request  decisions  and  (2)  capture  the  cost  and  schedule 
impacts  of  each  proposed  change  in  the  Navy  ERP  automated  change 
control  tracking  tool. 

To  increase  the  value  of  Navy  ERP  IV&V,  we  recommend  that  the 
Secretary  of  Defense  direct  the  Secretary  of  the  Navy,  through  the 
appropriate  chain  of  command,  to  (1)  stop  performance  of  the  IV&V 
function  under  the  existing  contract  and  (2)  engage  the  services  of  a  new 
IV&V  agent  that  is  independent  of  all  Navy  ERP  management, 
development,  testing,  and  deployment  activities  that  it  may  review.  In 
addition,  we  reiterate  our  prior  recommendation  relative  to  ensuring  that 
the  Navy  ERP  IV&V  agent  report  directly  to  program  oversight  officials, 
while  concurrently  sharing  IV&V  results  with  the  program  office. 
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Agency  Comments 

In  written  comments  on  a  draft  of  this  report,  signed  by  the  Assistant 
Deputy  Chief  Management  Officer  and  reprinted  in  appendix  II,  the 
department  concurred  with  our  recommendations,  and  stated  that  it  will 
take  the  appropriate  corrective  actions  within  the  next  7  months. 

We  are  sending  copies  of  this  report  to  interested  congressional 
committees;  the  Director,  Office  of  Management  and  Budget;  the 
Congressional  Budget  Office;  and  the  Secretary  of  Defense.  The  report 
also  is  available  at  no  charge  on  our  Web  site  at  http://www.gao.gov. 

If  you  or  your  staffs  have  any  questions  on  matters  discussed  in  this 
report,  please  contact  me  at  (202)  512-3439  or  hiter@gao.gov.  Contact 
points  for  our  Offices  of  Congressional  Relations  and  Public  Affairs  may 
be  found  on  the  last  page  of  this  report.  GAO  staff  who  made  major 
contributions  to  this  report  are  listed  in  appendix  III. 


Randolph  C.  Hite 
Director 

Information  Technology  Architecture 
and  Systems  Issues 


Page  24 


GAO-09-841  DOD  Business  Systems  Modernization 


Appendix  I:  Objectives,  Scope,  and 
Methodology 


Our  objectives  were  to  determine  whether  (1)  system  testing  is  being 
effectively  managed,  (2)  system  changes  are  being  effectively  controlled, 
and  (3)  independent  verification  and  validation  (IV&V)  activities  are  being 
effectively  managed  for  the  Navy  Enterprise  Resource  Planning  (ERP) 
program. 

To  determine  if  Navy  ERP  testing  is  being  effectively  managed,  we 
reviewed  relevant  documentation,  such  as  the  Test  and  Evaluation  Master 
Plan  and  test  reports  and  compared  them  with  relevant  federal  and  related 
guidance.  Further,  we  reviewed  development  test  plans  and  procedures 
for  each  test  event  and  compared  them  with  best  practices  to  determine 
whether  well-defined  plans  were  developed.  We  also  examined  test  results 
and  reports,  including  test  readiness  review  documentation  and  compared 
them  against  plans  to  determine  whether  they  had  been  executed  in 
accordance  with  the  plans.  Moreover,  to  determine  the  extent  to  which 
test  defect  data  were  being  captured,  analyzed,  and  reported,  we  inspected 
80  randomly  selected  defects  from  a  sample  of  2,258  defects  in  the 
program’s  test  management  system.  In  addition,  we  reviewed  the  history 
logs  associated  with  each  of  these  80  defects  to  determine  whether 
appropriate  levels  of  control  were  in  place  to  ensure  that  any  changes  to 
the  results  were  fully  documented.  This  sample  was  designed  with  a  5 
percent  tolerable  error  rate  at  the  95  percent  level  of  confidence,  so  that,  if 
we  found  0  problems  in  our  sample,  we  could  conclude  statistically  that 
the  error  rate  was  less  than  4  percent.  In  addition,  we  interviewed 
cognizant  officials,  including  the  program’s  test  lead  and  the  Navy’s 
independent  operational  testers,  about  their  roles  and  responsibilities  for 
test  management. 

To  determine  if  Navy  ERP  changes  are  being  effectively  controlled,  we 
reviewed  relevant  program  documentation,  such  as  the  change  control 
policies,  plans,  and  procedures,  and  compared  them  with  relevant  federal 
and  industry  guidance.  Further,  to  determine  the  extent  to  which  the 
program  is  reviewing  and  approving  change  requests  according  to  its 
documented  plans  and  procedures,  we  inspected  60  randomly  selected 
change  requests  in  the  program’s  configuration  management  system.  In 
addition,  we  reviewed  the  change  request  forms  associated  with  these  60 
change  requests  and  related  control  board  meeting  minutes  to  determine 
whether  objective  criteria  for  considering  a  proposed  change,  including 
estimated  cost  or  schedule  impacts,  were  adopted.  In  addition,  we 
interviewed  cognizant  officials,  including  the  program  manager  and 
systems  engineer,  about  their  roles  and  responsibilities  for  reviewing, 
approving,  and  tracking  change  requests. 
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Appendix  I:  Objectives,  Scope,  and 
Methodology 


To  determine  if  IV&V  activities  are  being  effectively  managed  we  reviewed 
Navy  ERP’s  IV&V  contract,  strategy,  and  plans  and  compared  them  with 
relevant  industry  guidance.  We  also  analyzed  the  contractual  relationships 
relative  to  legal  standards  that  govern  organizational  conflict  of  interest.  In 
addition,  we  examined  IV&V  monthly  status  reports,  work  plans,  an 
assessment  report,  and  a  quarterly  report,  to  determine  the  extent  to 
which  contract  requirements  were  met.  We  interviewed  contractor  and 
program  officials  about  their  roles  and  responsibilities  for  IV&V  and  to 
determine  the  extent  to  which  the  program’s  IV&V  function  is 
independent. 

We  conducted  this  performance  audit  at  Department  of  Defense  offices  in 
the  Washington,  D.C.,  metropolitan  area;  Annapolis,  Maryland;  and 
Norfolk,  Virginia;  from  August  2008  to  September  2009,  in  accordance  with 
generally  accepted  government  auditing  standards.  Those  standards 
require  that  we  plan  and  perform  the  audit  to  obtain  sufficient,  appropriate 
evidence  to  provide  a  reasonable  basis  for  our  findings  and  conclusions 
based  on  our  audit  objectives.  We  believe  that  the  evidence  obtained 
provides  a  reasonable  basis  for  our  findings  and  conclusions  based  on  our 
audit  objectives. 
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Appendix  II:  Comments  from  the  Department 
of  Defense 


DEPUTY  CHIEF  MANAGEMENT  OFFICER 
9010  DEFENSE  PENTAGON 
WASHINGTON,  DC  20301-9010 


$UG  Zl  2003 

Mr.  Randolph  C.  Hite 

Director,  Information  Technology  Architecture  and  Systems  Issues 
U.S.  Government  Accountability  Office 
441  G  Street,  N.W. 

Washington.  DC  20548 


Dear  Mr.  Hite: 

This  is  the  Department  of  Defense  (DoD)  response  to  the  GAO  draft  report 
09-841,  “DOD  BUSINESS  SYSTEMS  MODERNIZATION:  Navy  Implementing  A 
Number  Of  Key  Management  Controls  On  Enterprise  Resource  Planning  System,  but 
Improvements  Still  Needed,1'  dated  July  21,  2009  (GAO  Code  3 10666). 

The  Department  concurs  with  all  four  of  GAO's  recommendations.  The  Navy 
Enterprise  Resource  Planning  (ERP)  program  office  will  take  the  appropriate  corrective 
actions  within  the  next  seven  months.  Detailed  responses  to  each  recommendation  are 
attached. 

We  appreciate  the  support  of  GAO  as  the  Department  further  advances  in  its 
business  transformation  efforts,  and  look  forward  to  continuing  our  partnership  in 
achieving  our  shared  goals. 

Elizabeth  A.  McGrath 

Assistant  Deputy  Chief  Management  Officer 

Attachment(s): 

As  stated 
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Appendix  II:  Comments  from  the  Department 
of  Defense 


GAO  DRAFT  REPORT  DATED  JULY  21 . 2009 
GAO-09-841  (GAO  CODE  310666) 


“DOD  BUSINESS  SYSTEMS  MODERNIZATION:  NAVY 
IMPLEMENTING  A  NUMBER  OF  KEY  MANAGEMENT  CONTROLS  ON 
ENTERPRISE  RESOURCE  PLANNING  SYSTEM,  BUT  IMPROVEMENTS 
STILL  NEEDED” 

DEPARTMENT  OF  DEFENSE  COMMENTS 
TO  THE  GAO  RECOMMENDATIONS 


RECOMMENDATION  1:  The  GAO  recommends  that  the  Secretary  of  Defense  direct 
the  Secretary  of  the  Navy,  through  the  appropriate  chain  of  command,  to  revise  the  Navy 
Enterprise  Resource  Planning  (ERP)  procedures  for  controlling  system  changes  to 
explicitly  require  that  a  proposed  change’s  life  cycle  cost  impact  be  estimated  and 
considered  in  making  change  request  decisions.  (Page  3 1/GAO  Draft  Report) 

DOD  RESPONSE:  Concur.  The  Navy  ERP  program  office  will  revise  the  Enterprise 
Change  Request  Process  and  Procedures  document  to  require  explicitly  that  a  proposed 
change’s  life  cycle  cost  impact  be  estimated  as  part  of  the  change  control  decision 
process.  Corrective  actions  to  address  GAO’s  recommendation  will  be  taken  by  the 
beginning  of  Fiscal  Year  (FY)  2010. 

RECOMMENDATION  2:  The  GAO  recommends  that  the  Secretary  of  Defense  direct 
the  Secretary  of  the  Navy,  through  the  appropriate  chain  of  command,  to  capture  the  cost 
and  schedule  impacts  of  each  proposed  change  in  the  Navy  ERP  automated  change 
control  tracking  tool.  (Page  3 1/GAO  Draft  Report) 

DOD  RESPONSE:  Concur.  The  Navy  ERP  program  office  will  update  its  automated 
change  control  tracking  tool  to  capture  “dollarized”  cost  impacts  and  better  identify 
schedule  impacts  of  future  proposed  changes.  Corrective  actions  to  address  GAO’s 
recommendation  will  be  taken  by  the  beginning  of  FY  2010. 

RECOMMENDATION  3 :  The  GAO  recommends  that  the  Secretary  of  Defense  direct 
the  Secretary  of  the  Navy,  through  the  appropriate  chain  of  command,  to  stop 
performance  of  the  Independent  Verification  and  Validation  (1  V&V)  function  under  the 
existing  contract.  (Page  3 1/GAO  Draft  Report) 

DOD  RESPONSE:  Concur.  The  Navy  ERP  program  office  plans  to  stop  IV&V 
functions  under  the  existing  contract  at  the  end  of  the  current  fiscal  year  (2009). 
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Appendix  II:  Comments  from  the  Department 
of  Defense 


RECOMMENDATION  4:  The  GAO  recommends  that  the  Secretary  of  Defense  direct 
the  Secretary  of  the  Navy,  through  the  appropriate  chain  of  command,  to  engage  the 
services  of  a  new  IV&V  agent  that  is  independent  of  all  Navy  ERP  management, 
development,  testing,  and  deployment  activities  that  it  may  review.  (Page  31/GAO  Draft- 
Report) 

POD  RESPONSE:  Concur.  The  Navy  ERP  program  plans  to  execute  future  IV&V 
functions  using  contract  support  that  is  not  associated  with  any  of  the  other  Navy  ERP 
program  activities  to  ensure  there  is  no  conflict  of  interest,  real  or  perceived.  Corrective 
actions  to  address  GAO’s  recommendation  will  be  taken  no  later  than  March  2010. 


2 


Attachment  1 
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Appendix  III:  GAO  Contact  and  Staff 
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GAO’s  Mission 

The  Government  Accountability  Office,  the  audit,  evaluation,  and 
investigative  arm  of  Congress,  exists  to  support  Congress  in  meeting  its 
constitutional  responsibilities  and  to  help  improve  the  performance  and 
accountability  of  the  federal  government  for  the  American  people.  GAO 
examines  the  use  of  public  funds;  evaluates  federal  programs  and  policies; 
and  provides  analyses,  recommendations,  and  other  assistance  to  help 
Congress  make  informed  oversight,  policy,  and  funding  decisions.  GAO’s 
commitment  to  good  government  is  reflected  in  its  core  values  of 
accountability,  integrity,  and  reliability. 

Obtaining  Copies  of 
GAO  Reports  and 
Testimony 

The  fastest  and  easiest  way  to  obtain  copies  of  GAO  documents  at  no  cost 
is  through  GAO’s  Web  site  (www.gao.gov).  Each  weekday  afternoon,  GAO 
posts  on  its  Web  site  newly  released  reports,  testimony,  and 
correspondence.  To  have  GAO  e-mail  you  a  list  of  newly  posted  products, 
go  to  www.gao.gov  and  select  “E-mail  Updates.” 

Order  by  Phone 

The  price  of  each  GAO  publication  reflects  GAO’s  actual  cost  of 
production  and  distribution  and  depends  on  the  number  of  pages  in  the 
publication  and  whether  the  publication  is  printed  in  color  or  black  and 
white.  Pricing  and  ordering  information  is  posted  on  GAO’s  Web  site, 
http://www.gao.gov/ordering.htm. 

Place  orders  by  calling  (202)  512-6000,  toll  free  (866)  801-7077,  or 

TDD  (202)  512-2537. 

Orders  may  be  paid  for  using  American  Express,  Discover  Card, 

MasterCard,  Visa,  check,  or  money  order.  Call  for  additional  information. 

To  Report  Fraud, 
Waste,  and  Abuse  in 
Federal  Programs 

Contact: 

Web  site:  www.gao.gov/fraudnet/fraudnet.htm 

E-mail:  fraudnet@gao.gov 

Automated  answering  system:  (800)  424-5454  or  (202)  512-7470 

Congressional 

Relations 

Ralph  Dawn,  Managing  Director,  dawnr@gao.gov,  (202)  512-4400 

U.S.  Government  Accountability  Office,  441  G  Street  NW,  Room  7125 
Washington,  DC  20548 

Public  Affairs 


Chuck  Young,  Managing  Director,  youngcl@gao.gov,  (202)  512-4800 
U.S.  Government  Accountability  Office,  441  G  Street  NW,  Room  7149 
Washington,  DC  20548 
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